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ABSTRACT 
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for educational change are examined; (1) targets for change, (2) 
techniques for monitoring change, (3) time for change, and (4) 
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RESEAECH METHODOLOGY FOR EDUCATIONAL CHANGE 
Samuel Messick 
Educational Testing Service 

In any consideration of research methodology for educational change, two 
basic questions loom large at the outset--namely, what is it that must be 
changed? and along which dimensions and in which direction should the 
change take place? A third important question concerning how the change is 
to be produced usually receives most of the attention, but often largely as 
a practical problem in the engineering of change, an approach that tends to 
be more concerned with means and procedures than it is with either of the 
other two issues --that is, either with the nature of the thing to be changed 
or with the goals or intended outcomes of the change process. 

A research methodology provides a systematic way of looking at a 
problem, a mode of inquiry for eliciting information and evidence leading 
to understanding and problem solution. Rather than spending the brief time 
allotted here discussing particular methods or techniques of inquiry and 
analysis that might be useful in fostering educational change, let us in- 
stead focus upon some of the critical problems in the production of change 
and see what general methodological strategies or "ways of looking" seem 
called for. First, let us consider the nature 'of the base from which change 
is to be mounted — the nature of the educational arena, as it were, especially 
those features that are typically targeted for change and those features 
that tend to impede attempts at alteration. Then, we shall turn to a con- 
sideration of the basis for change, of the sources of the ideas underlying' 
new plans and new goals for the educational enterprise. 
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Targets for Change 

One of the most critical features of the educational arena is that it 
constitutes a system in the technical sense of a complex set of elements 
that functions as a whole by virtue of the interdependence of its parts.^ 

There are many pai’ties to the system- -such as teachers, students, parents, 
principals, counsellors, communities, superintendents, school boards--and 
many parts and processes. 

In one sense the system is hi^ly dynamic, in that particular individuals 
involved frequently change or are replaced. Individual students flow through 
the various grades or levels of the system, for example, teachers are trans- 
ferred or promoted, and some communities have such a high rate of populabion 
exchange that not only is considerable student replacement introduced within 
grade level but the community's social characteristics are gradually trans- 
formed. It seems likely that strategies of intervention and change will 
have to vary depending upon the rates of such population exchanges --that 
is, depending upon the degree of interdependence between particular indi- 
viduals and the organization or system. 

In another sense the system is hi^ly stable, in that it displays a 
huge number of programmatic and behavioral regularities. Most programmatic 
regularities are intended to produce or to change behavioral regularities. 

For example, the inexorable programmatic regularity that confronts each 
student on every school day of every school year with numbers and mathematics 
is presumably intended to increase in cumulative fashion certain behavioral 
regiilarities in the student, as reflected in his repertoire for quantitative 
thinking and his level of mathematical mastery. The frequency and level 
or quality of these behavioral regularities, in turn, offer important 
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criteria for Judging the extent to which the intended outcomes are being 
realized. 

Not all of the existing behavioral regularities of students, teachers, 
or other participants in the system are intended outcomes or processes, 
however. Many of them reflect extraneous consistencies, while others 
represent unintended outcomes or side effects, which may be either 
desirable or undesirable. A finding, for example, that teachers ask 
questions in class at a very high rata, about 50 times as freq^uently as 
students do, and that a very large proportion of these questions require 
straight recall from the students is a behavioral regularity, to be sure, 
but is it an intended or a desirable state of affairsV 

Since most of the impetus for educational change stems from someone 
noticing a regularity that he considers undesirable (or the absence of a 
regularity he considers desirable), it is nob surprising that most efforts 
at change attack the offending regularity frontally. Failure to appreciate 
that the regularity is embedded in a system of interdependencies has double- 
edged consequences. To begin with, the successful modification of a part 
of the system may have unintended and possibly adverse effects in other 
parts of the system, thereby not improving matters very much overall. 
Furthermore, if the network of interdependencies supporting a regularity is 
overlooked, it cannot very well be countered, so there is little likelihood 
that circumscribed alterations will be successfully maintained in the face 
of inertial pressures toward reabsorption of the changed part back into the 
old pattern. The more things change, the more they stay the same. And the 
situation is even more intransigent than this, for we are dealing not only 
with a system but with a culture . The existing regularities are determined 
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and maintained by roles and expectations, attitudes and values, tradition 
and history. Under these circumstances, effective and long-range educa- 
tional change requires not only individual change and program change but 
institutional change. 

Let us briefly consider a case history of a recent intensive effort 
to modify some perceived undesirable regularities. The case concerns the 
introduction of the nev/ math into the school curriculum. In reporting the 
results of a recent observational study, Sarason (l97l) summarized the 
situation as follows: 

...for some time before the first Russian Sputnik in 1957 
there was a good deal of dissatisfaction with the teach- 
ing of mathematics in the public schools . The leadership 
of this dissatisfaction was primarily in the universities, 
and the content of the criticism took different forms . 

B\it on at least one point there was complete agreement — 
the way children were being tau^t math was an unmitigated 
bore and disaster that very few children could survive 
either in the sense that they experienced the jcy of the 
world of numbers or pursued mathematj.es as a career. The 
Russian Sputnik catalyzed the effort to change the teach- 
ing of math, and various new maths were developed in uni- 
versity centers and introduced into the schools. After 
several years of the new math we observed. .. (its teaching) 
in a number of classrooms in several school systems. . . .joy 
is the last word in the English language that one could 
apply to the children in those classrooms ... .If our obser- 
vations and those of others have validity and generality, 
one would have to predict that the goal of more and better 
mathematicians and scientists .. .will not be met. If so, 
we will have another sad example of how the more things 
change the more they remain the same [pp. 19 & ^6]. 

As Sarason (l97l) points out, this conclusion is perhaps too generous, 
for not very much has basically changed at all. Why? To begin with, the 
original diagnosis of the problem focused upon a particular set of undesira- 
ble behavioral regularities and their presumed source in the programmatic 
regularities of the traditional mathematics curriculum; change was to be 
brought about through substituting a new set of regularities by means of 
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a new curriculum. Very little was said about teachers or the teaching 
process, nor was the problem formulated in such a way as to require the 
‘ involvement of teachers as an integral part of the change effort. Further- 

more, the impetus for change came primarily from outside the school setting, 

f 

and the new materials were introduced into the schools, in many cases im- 
posed upon them, with little attention paid to the existing institutional 
culture or its social and psychological concomitants. 

In the settings observed, teachers were trained in the new math in 
summer workshops in highly traditional fashion as if this were a standard 
problem of imparting knowledge and developing skills, with little recog- 
nition given to the difficulties entailed in having to unlearn highly over- 
learned ways of thinking at the same time that new modes of conceptualizing 
were to be acquired. The teachers had unanticipated difficulty in learning 
the new material to a degree comparable to their grasp of the old curriculum, 
« which itself was hardly at a level that could be characterized as decisive 

mastery- -it was typically more like an uneasy truce achieved after years 
of struggle. They were obliged to embark on the new enterprise with very 
little supervision — nowhere near enough to bolster the tentative under- 
standing or dispel the attendant apprehensiveness. Nonetheless, they 
settled down, as is their wont, to teach in the way in which they themselves 
had been taught. 

One consequence of such an. emphasis upon materials and curriculum 
content is that all those far-reaching goals and intended outcomes concern- 
ing the development of positive affect and thinking skills and manpower 
resources are lost sight of, and the development and delivery of the means 
becomes- a goal'.in' itself . The whole enterprise then tends to be evaluated 
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in terms of the successful accomplishment of the engineering objective of 
substituting one set of textbooks for ano'ther. 

There are several implications of all this for research methodology, 
and one of them relates to the formulation of the problem: The question of 

how change is to be produced is not only an engineering problem but a research 
problem. The process of change in each instance should be conceptualized 
in such a way that it appropriately takes into account the complexities and 
dimensions of differentiation of the cu3,tural setting in which change is to 
occur. This means that we need to develop procedures for uncovering the 
relevant dimensions and relationships, as well as alternative techniques 
for dealing with them. Since it is unlikely that proposed change will have 
the same significance for »-1 1 of the different parties ajiid groupings involved 
in the setting, the social forces surrounding change are likely to be multi- 
plex. The change process, then, must be di.fferentiated and flexible to be 
able to capitalize upon existing sources of support, to recruit additional 
support as needed, or to confront the forces of opposition. 

The catch in all this is that we need a considerable amount of in- 
formation about existing regularities and social forces in order to con- 
ceptualize the change process realistically in the first place. The usual 
reaction to this state of affairs is to urge the external advocate of 
change to immerse himself j,n the situation to get a feel for the inter- 
acting forces and vested interests before attempting specific interventions. 
This is all well and good, but it takes time and is rarely done systemati“. 
cally enough to permit generalization to other settings or even to provide 
a verifiable data base to substantiate ’what often turns out to be a private 
view of system functioning. Another taok is to encoiirage change from within 
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the system by participants who know the territory well. But there is no 
guarantee that participation per se will provide a clear or unbiased view — 
quite the contrary. Nor can we assume any intrinsic relationships between 
familiarity with the setting and one's conception of the change process or 
of the requirements of experimental intervention. Still another approach 
is to move outside the existing system and attempt to construct a new 
system under less constrained circumstances. The hope here is that a 
parallel system offering clear demonstrations of exemplary practice will 
somehow move the old systems to emulate it, but there are vast differences 
between creating a new system and changing an existing one. 

Techniques for Monitoring Change 

What we need is a systematic basis for understanding the system. We 
need methodologies for objectively determining and describing basic charac- 
teristics of system functioning and for evaluating both the changes in those 
characteristics and the efficacy of the change process in relation to intended 
and unintended outcomes of the system and of intervention programs . 

One possibility might be to introduce into school systems a continuous 
program of information collection and analysis to uncover existing consis- 
tencies and structu’^es and to monitor their perturbations over time. Such 
an information and assessment program would have to be comprehensive enough 
to assess a wide range of student outcomes representing intended and unin- 
tended consequences of the educational system, as well as characteristics 
of teachers, students, administrators, programs, and settings that might 
interact with each other to produce differential results. It would also be 
particularly important to include some provision for observing and documenting 
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those most critical of all regularities occurring between teacher and student 
in the classroom. Given the cultural basis of much system functioning, it 
would be vital to incorporate procedures for assessing the attitudes and 
values, roles and relationships, perceptions and expectations, and aspira- 
tions and goals of the various parties to the system and to attempt to 
unravel dimensions of consensus and contention permeating the social matrix. 
With such a multiplicity of components, it is obvious that the program 
would have to include an analytical methodology capable of multivariate, 
longitudinal, and interactional comparisons to provide mechanisms for 
disentangling the threads of multiple covariation. Ultimately such an 
assessment effort should attempt to go beyond the description of system 
functioning as it exists to the development of causal models relating prior 
conditions and processes to outcomes , perhaps through the use of computer 
simulation techniques, so that potential consequences of alternative action 
programs could be systematically anticipated. 

An intervention project could be introduced into an educational system 
being monitored in this fashion with prior warning about existing social 
pressures and with periodic feedback enabling adjustments of the change 
process. The impact of the intervention could then be systematically 
evaluated, in strai^t forward if not routine manner, in terms of changes 
in measures of behavioral regularities, especially student outcomes. 

Such a comprehensive assessment program is an ideal that would require 
considerable time and effort to realize. .It offers the advantages of a 
wide range of information bearing on administrative and instructional 
decision making, evaluation, and accountability, and in the long run could 
serve as a vehicle for continuing research on the functioning of educational 



systems. As a continuing program, it would also provide an early warning 
system signaling the appearance of new types of students and new or changing 
conditions, thereby permitting timely modifications in system functioning to 
accommodate to new inputs and circumstances. Its major disadvantages are 
the time and resources required to develop, implement, and maintain it. 

There are several alternatives to this strategy, of course,, and one 
of them is of particular interest because of its responsiveness to the time 
pressures for change. The foregoing research strategy gains much of its 
power for evaluating and understanding change by introducing interventions 
in the course of an information collection program. The alternative to be 
considered next reverses the sequence and introduces information collection 
in the course of an intervention program. 

Time for Change 

One of the most serious difficulties in developing acceptable research 
methodologies for educational change is the widespread adoption of a con- 
sistently warped time perspective for viewing change. Over and over again, 
with exasperating consistency, the time required to initiate real change 
and to cumulate lasting effects is woefully underestimated. As a result, 
much educational planning is based upon relatively brief eiq)eriences with 
programs, and decisions are frequently made to initiate programs — 'or to 
terminate them- -on the basis of short-term evidence, usually obtained from 
ad hoc studies. This can be particularly devasting when a: promising 
developmental program is subjected to summative evaluation prematurely. 

Although there are inherent dangers in making long-range policy 
decisions on the basis of short-term research, in a time of pressing social 
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problems and rapid social change the alternative of basing such policy 
decisions on long-term research does not appear to be particularly viable 
either. The problems are simply too pressing and the conditions changing 
too rapidly for us to postpone action pending the results of long-term 
research and development efforts that might increase the likelihood of 
widespread positive effects . Action is needed now. But neither can we 
afford the wasted time and resources of inadequately researched programs 
that turn out to be ineffective. There are strong pressures not only for 
action but for accountability. 

One strategy that appears to offer some promise in the face of such 
conflicting pressures is to do both things at once--both initiate the 
action program and conduct research on the effectiveness and consequences 
of the action at the same time. Specifically, this approach calls for the 
undertaking of evaluative research on a continuous basis as an integral part 
of intervention program efforts. That is, once the action program is pro- 
visionally formulated on the basis of the best available knowledge and ; 
previous research, we would then proceed to carry out both the program and 
the evaluative research simultaneously. This would be accomplished by 
including within the administration of the program provision for collecting 
information relevant to its evaluation and improvement. The level of the 
research effort, of course, should be appropriate to the magnitude of the 
program, to the soundness of the prior basis for the program, and to the . 
potential consequences of program failure. There would thus be no delay 
in program implementation, but at the same time the research would be as 
long-range as the program was . 

. By periodically monitoring program effectiveness, this continuous 
evaluative research would process data bearing directly on questions of 
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program evaluation and accountability. But it would also provide information 
relevant for improving the program, as well as for modifying the program to 
accommodate to new types of students and new or changing conditions. Such 
continuous evaluative research would thus provide at the program level a 
rational basis for adaptive action (Messick, 1970; Messick & Barrows, 1972). 

Testaments for Change 

Let us now turn to questions of the source and direction of change, of 
the origin of ideas for new plans and new aspirations in education. At first 
glance it might appear that in a rational enterprise like education these 
ideas derive fiom a careful examination of evidence, that they represent 
direct implications of the findings- of research. But let us analyze that 
notion for a moment. Consider, for example, a body of evidence indicating 
that a large part of man's behavior is shaped and maintained by its conse- 
quences, which are under the control of environmental contingencies. 

Consider next a social or educational theorist like Skinner (1971) 
having a particular conception about the nature of man--"not of a body with 
a person inside, but of a body which is a person in the sense that it dis- 
plays a complex repertoire of behavior [p. I99l • " He might reason from such 
evidence somewhat as follows; , Since man is controlled by his environment 
and the environment can be manipulated, we shoiold use a technology of 
behavior to engineer a society having -sufficient- controlling contingencies 
to maintain socially desirable and productive behavior. If this means 
that autonomous man, the inner man, is being abolished, "That is a step 
forward," according to Skinner (l971^ p. 215). "His abolition has long 
been overdue," he maintains,; for "only by dispossessing him can we turn to 
the real causes of human behavior [p. 200-201] ." 
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Now consider another theorist viewing the same data but with a 
different conception of the nature of man-- "To say that a man is a person 
is to say that in the depth of his being he is more a whole than a part 
and more independent than servile.” Such a theorist would reason very 
differently. He might see behavior control by environmental contin- 
gencies as a problem for education, not a solution, and strive to bring 
the operation of those contingencies into the awareness of students in 
an attempt to reduce the preemptiveness of conditioned behavior. We 
mi^t then create, by a process more akin to societal guidance than to 
social engineering (Etzioni, 1968, 1970); a society that is not beyond 
freedom and dignity, one with enough alternative opportunities to 
challenge the individual to choose, freeOy and responsibly, those 
environmental contingencies that will come to maintain his behavior. 

In this kind of society, autonomous man lives on, for. ’’freedom is found 
in that kind of interaction which maintains an environment in which 
human desire and choice count for something [Dewey, 1922].’’ 

What is at issue in this example is not the research findings, 
although there is indeed controversy over the consistency of results in 
this, area with human subjects. What is at issue is ideology. It is not 
the implications of research results per se that are to be implemented 
in the proposed strategies, it is the implications of research as inter- 
preted or filtered through a particular ideology about the nature of man 
and society. In this sense, research does not directly determine the aims 
of educational practice or educational change, nor should we expect it to.. 
It instead serves to refine, to justify, and in its finest moments to 
challenge directions that are primarily, ideologically determined. Its 
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most powerful impact comes on those rare occasions when it stimulates a 

change in the mediating conceptions , especially when it produces a change 

. 5 

in our conception of the human being as a learner. 

Several such conceptual changes have occurred in education in the past 
fifty years or so, and serious attempts were made to implement the impli- 
cations of the new conceptions in practice— as when we shifted from the 
notion of the learner as an empty organism to that of the learner as a 
dynamic organism, to that of the learner as a social organism, to that of 
the learner as an inquisitive, stimulus -seeking organism, etc. One of 
these conceptual changes occurred fairly recently in early childhood 
education when research with very yoimg children indicated that they’ 
can indeed learn cognitive skills and meaningfully process symbolic and 
semantic information, thereby shaking our general ass'omptions about the 
maturational limitations of young children and their readiness for system- 
atic educational experiences prior to some magical "schoo;i age." 

The history of these major conceptual shifts is not particularly 
encouraging, however, for they tend to represent fairly global reformula- 
tions in viewpoint while the phenomena they refer to almost certainly 
require more differentiated treatment . The human learner may indeed 
function as a social organism, to be sure, but might he not also function 
as a dynamic organism as well? And as an inquisitive organism? And even 
as a partly-empty organism? And more as one than another at different times 
and under different circumstances? Historically, attempts to implement the 
implications of each new viewpoint have tended to be relatively single- 
minded, resulting not only in marked pendulum swings in practice but 
pendulum swings in different directions. There has thus been considerable 
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change on the educational scene over the years but not necessarily very much 
progress. 

Given the central role of ideology in determining aims and action in 
education, it vfould seem imperative that we undertake systematic analyses 
of how ideological dispositions affect .action through the medium of judg- 
ment and decisionmaking (Dewey, 1964 j Stake, 1970). This would req'uire 
measurement methodologies, perhaps techniques of multivariate analysis or 
multidimensional scaling, for assessing consistent individual viewpoints 
about the nature of man and the aims of education and society and for 
developing an articulated description of each viewpoint in terms of the 
structure of interrelationships among its component values (Jackson & . 

Messick, 1965^ Tomkins', 1965 ^ '•Tucker, )& Mess ic.k, 1965; Uhl, 1971)* An 
understanding of the structure of different value systems would be partic- 
ularly important in attempting to trace their interaction in areas of„_---rrrr^, 
consonance and confrontation. Different ideologies may exhibit substantial: 
degrees of overlap in shared values, for example, while a particular ideology 
may contain potentially conflicting value components that do not always lead 
to mutually supporting implications. The personality and environmental 
correlates of these viewpoints could also be examined for clues as to their 
origins and the conditions under which they aire likely to predorainate . 

With this kind of strategy, we might then be able to relate differences 
in educational objectives to characteristics of decision makers, their 
underlying values, and their situational constraints. By evaluating the 
empirical consequences of action programs in terms of structures of inter- 
related values instead of lists of objectives, we would also be better able 
to appraise in systematic fashion the inport of unintended as well as 
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intended outcomes. 
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To recapitulate briefly, it. looks g-s. if the nature cof the thing to be 
changed, the educational arena, is not only a complex system composed of 
many constituents but a complex culture comprising multiple roles and 
pluralistic values, and that if we are to understand the functioning of 
that system — or to change it — we must take into account the interplay of 

f 

those jroles and values in supporting (or subverting) system regularities. 
We have seen further that the sources and direction of change are also 
primarily a function of value perspectives and that individual and social 
values must, therefore, play a central role — often implicitly, to be sure- 
in educational planning and policy making. The implications of these 
points for research methodology seem clear. We need methodologies that 
no longer strive after Max Weber's perverse ideal of a value-free social 
science, but rather methodologies that are openly value-centered (Gouidner 

1962). 

In short, although the methods of empirical social science research 
are typically employed to guide means -judgments, usually by investigating 
effectiveness and efficiency, the proposed strategy calls for the applica- 
tion of social science methodology to help understand ends- judgments and 
to clarify and improve means -ends decisions. If this approach doesn't 
exactly make valuation a social science, it would at least make the 
educational applications of social science more valuative (Dewey, 1970 ; 
Edel, 1970). . • ■ 
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